Summary

The processor previously retried blob NotFound responses indefinitely. process.Subscribe returned nil from its handler without acking, so NATS redelivered the same orphaned message forever — generating hundreds of thousands of GetObject errors per processor over weeks for blobs that no longer exist.

After maxBlobNotFoundRetries (3) consecutive NotFound responses for a message, the message is now acked to stop redelivery. Other read errors retain their previous redelivery behavior.

How retries are counted

When the underlying transport exposes a delivery count, that is preferred so the threshold is shared across replicas and survives processor restarts:

• NATS JetStream: uses Metadata().NumDelivered via gocloud.dev/pubsub.Message.As(&jetstream.Msg).
• Other backends (incl. mempubsub in tests): falls back to a per-key in-process counter.

A Prometheus counter guac_processor_blob_notfound_dropped_total (label processor_id) is registered for alerting on orphan accumulation, and a structured warn log is emitted on each drop with the blob key and attempt count.

What this does not address

Why orphan events get published in the first place (likely TTL mismatch between blob lifecycle and NATS retention, or out-of-band cleanup). That belongs in a separate investigation.

Test plan

Run on this branch:

• go build ./... clean (exit 0)
• go vet ./... clean (exit 0)
• go test -race -timeout=60s ./... -count=1 — 79 packages OK, 0 fail, 0 data races. Note: at the Makefile's default -timeout=30s under race-detector parallelism, the unrelated pkg/ingestor/parser/common/scanner package occasionally tips over (it runs ~25–30 s in isolation, with the same behaviour on main); raising the timeout to 60 s clears it.
• golangci-lint run ./pkg/handler/processor/process/... — 0 issues
• New unit tests pass: TestNotFoundTracker (per-key increment/clear isolation) and TestDeliveryCountSource (label values). Existing Test_ProcessSubscribe continues to pass unchanged.